ESQL: Refactor value reading so it can split Pages (#130573) #130784
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This refactors our `ValuesSourceReaderOperator` so it can split pages
when it reads large values. It does not *actually* split the pages as
that's a bit tricky. But it sets the stage for the next PR that will
do so.
* Move `ValuesSourceReaderOperator` to it's own package
* Move many inner classes into their own top level classes
* Extend from `AbstractPageMappingToIteratorOperator` instead of
`AbstractPageMappingToOperator`
* This allows returning more than one `Page` per input `Page`
* In this PR we still always return one `Page` per input `Page`
* Make new `ReleasableIterator` subclasses to satisfy
`AbstractPageMappingToIteratorOperator`
* Change `status` of loading fields from `pages_processed` to
`pages_received` and `pages_emitted`
* Fix a bug in `AbstractPageMappingToOperator` which can leak circuit
breaker allocation if we fail to during `receive`. This isn't possible
in the existing implementations but is possible
in `ValuesSourceReaderOperator`.
* Add a test with large text fields. Right now it still comes back in
one page because we don't cut the pages.
Closes elastic#130727
|
I'll merge #130838 into this to fix the double-close issues we're seeing. |
Fixes a bug during field loading where we could double-close blocks if we failed to allocate memory during the un-shuffling portion of field loading from single segments. Unit test incoming in the followup. Closes elastic#130426 Closes elastic#130790 Closes elastic#130791 Closes elastic#130792 Closes elastic#130793 Closes elastic#130270 Closes elastic#130788 Closes elastic#130122 Closes elastic#130827
…l_danger_zone_8_19
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This refactors our
ValuesSourceReaderOperatorso it can split pages when it reads large values. It does not actually split the pages as that's a bit tricky. But it sets the stage for the next PR that will do so.ValuesSourceReaderOperatorto it's own packageAbstractPageMappingToIteratorOperatorinstead ofAbstractPageMappingToOperatorPageper inputPagePageper inputPageReleasableIteratorsubclasses to satisfyAbstractPageMappingToIteratorOperatorstatusof loading fields frompages_processedtopages_receivedandpages_emittedAbstractPageMappingToOperatorwhich can leak circuit breaker allocation if we fail to duringreceive. This isn't possible in the existing implementations but is possible inValuesSourceReaderOperator.Closes #130727